feat: add cache_messages to bedrock model #1029
Conversation
DayuanJiang
requested a deployment
to
manual-approval
GitHub Actions
Waiting
src/strands/models/bedrock.py
Outdated
| cleaned_messages.append({"role": message["role"], "content": cleaned_content}) | ||
| else: | ||
| cleaned_messages.append(message) | ||
| return cleaned_messages |
There was a problem hiding this comment.
Can you explain the purpose of clearing cachePoints from previously processed messages?
There was a problem hiding this comment.
Thanks for pointing that out. Originally, the implementation would add a cache point to the last message on each call. To avoid accumulating too many cache points across multiple calls, I decided to remove all existing cache points before adding the new one.
However, I realized that this approach was problematic because it modified the original messages passed to the function. So I modified the code to create a copy of the messages and add the cache point to the copied version, which means clearing existing cachePoints is no longer necessary.
I have also updated the PR description to reflect this change.
DayuanJiang
requested a deployment
to
manual-approval
GitHub Actions
Waiting
|
Hi, we are working on #937 where we are introducing support for the more generic SystemContentBlock. I believe this will be a more general solution and we should wait before introducing this bedrock specific logic |
|
Hi @dbschmigelski, thanks for responding! I appreciate the heads up about #937, but I believe SystemContentBlock is specifically for handling system prompts. While both involve caching, they operate on different parts of the request:
These are complementary features rather than overlapping ones, so I don't think we need to wait for #937 to merge this. That said, happy to discuss if you see a specific conflict I am missing! |
3 participants
Description
Introduced the
cache_messagesconfiguration parameter toBedrockModelto enable message caching for intermediate steps.When
cache_messages = "default", the agent will add a cache point at the end of the message list each time it calls a Bedrock model. This allows the agent to cache intermediate steps effectively while preserving any existing cache points users may have set.Key implementation details:
format_requestRelated Issues
#404 #1015
Documentation PR
None
Type of Change
New feature
Testing
test_format_request_cache_messages(): Validates that existing cache points are preserved and a new cache point is added at the endtest_format_request_cache_messages_does_not_modify_original(): Verifies that the original messages passed to format_request are not mutatedI ran
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.